System and method for registration of imaging data

ABSTRACT

A method of obtaining imaging data from a tissue region is provided. The method is effected by associating imaging data with a surface contour of the tissue region and utilizing a model of contour changes in the tissue region to transform the data to reflect changes in the surface contour.

RELATED APPLICATIONS

This Application is a National Phase of PCT Patent Application No.PCT/IL2008/001683 having International filing date of Dec. 28, 2008,which claims the benefit of U.S. Provisional Patent Application No.61/006,223 filed on Dec. 31, 2007. The contents of the aboveApplications are all incorporated herein by reference.

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to systems and methods for enablingmedical imaging data registration and more particularly, to systems andmethods which enable registration of tissue imaging data for thepurposes of enhancing diagnostic resolution.

Medical imaging is routinely used to image the human body or portionsthereof for clinical or research purposes.

For over 70 years, medical imaging had almost exclusively depended onconventional film/screen X-ray imaging. However, in the last 40 years,medical imaging has experienced major technological growth which hasresulted in the development and commercialization of new imagingtechnologies. Such technologies, which include X-ray ComputedTomography, Magnetic Resonance Imaging, Digital Subtraction Angiography,ultrasound, thermography and nuclear emission imaging (e.g. PET, SPECT,etc.) are now routinely used in detection and diagnosis of disease.

The availability of such diagnostic technologies provides a physicianwith a range of diagnostic tools to choose from and also potentiallyenables correlation (registration) of several different imagingapproaches thus greatly enhancing accuracy of diagnosis.

Having a range of diagnostic tools to choose from can potentiallyenhance the ability of a physician to diagnose a disease, however, it isthe correlation of results from several imaging approaches which has thegreatest potential in enhancing diagnostic accuracy.

Although a patient can be subjected to multiple imaging approaches (e.g.x-ray and ultrasound), the images obtained are not easily registered orcorrelated with one another. Differences in scale, position, or in theorientation of the imaging plane are almost inevitable. With certaintissues (e.g. breast) imaging registration is further hampered bydeformation of the tissue which can result from the imaging technique(e.g. compression of breast tissue between mammography plates).

The prior art is replete with approaches for enabling registration ofmedical images, most requiring the use of orientation markers or modelswhich are typically constructed using 3-D imaging approaches (e.g. MRI).

Prior art registration approaches are typically designed for registeringimaging data obtained by x-ray, ultrasound or MRI. However, in the caseof thermographic imaging, such approaches are incapable of providing anaccurate registration since thermographic data is derived from thesurface of the imaged body portion rather than the internal tissues.

A thermographic image is typically obtained by collecting from the abody of the subject radiation at any one of several infrared wavelengthranges and analyzing the radiation to provide a two-dimensionaltemperature map of the surface. The thermographic image can berepresented as a visual image with or without corresponding temperaturedata. The output from infrared cameras used for infrared thermographytypically provides an image comprising a plurality of pixel data points,each pixel can provide relative temperature information which isvisually displayed, using a color code or grayscale code. Thisinformation can be further processed by computer software to generatefor example, mean temperature for the image, or a discrete area of theimage, by averaging temperature data associated with all the pixels or asub-collection thereof.

Since shifts in body temperature can indicate the presence of adisorder, (e.g. inflammation caused an increase in temperature), athermographic image can be used by a physician to determine whether ornot a site includes presence of a disorder.

While reducing the present invention to practice, the present inventorshave uncovered that surface contour data, especially when combined withthermal imaging data can be used for registration of imaging modalities.

SUMMARY OF THE INVENTION

The present invention provides a method of obtaining imaging data from atissue region comprising (a) obtaining a first surface contour of thetissue region in a first state (b) obtaining a first imaging data fromthe tissue region in the first state and associating it with the firstsurface contour (c) obtaining a second surface contour of the tissueregion in a second state; (d) using the first surface contour and thesecond surface contour to model the tissue region at the second state;and (e) transforming the first imaging data into a second imaging dataassociated with the tissue region in the second state.

According to preferred embodiments of the present invention, the imagingdata is thermal imaging data.

According to preferred embodiments of the present invention, the imagingdata is X-ray imaging data.

According to preferred embodiments of the present invention, the imagingdata is ultrasound imaging data.

According to preferred embodiments of the present invention,transforming the imaging data is effected by correcting an imaging planeof said first imaging data according to the model.

The present invention successfully addresses the shortcomings of thepresently known configurations by providing a simple and yet highlyeffective approach for registering imaging data.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although methods and materialssimilar or equivalent to those described herein can be used in thepractice or testing of the present invention, suitable methods andmaterials are described below. In case of conflict, the patentspecification, including definitions, will control. In addition, thematerials, methods, and examples are illustrative only and not intendedto be limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

The invention is herein described, by way of example only, withreference to the accompanying drawings. With specific reference now tothe drawings in detail, it is stressed that the particulars shown are byway of example and for purposes of illustrative discussion of thepreferred embodiments of the present invention only, and are presentedin the cause of providing what is believed to be the most useful andreadily understood description of the principles and conceptual aspectsof the invention. In this regard, no attempt is made to show structuraldetails of the invention in more detail than is necessary for afundamental understanding of the invention, the description taken withthe drawings making apparent to those skilled in the art how the severalforms of the invention may be embodied in practice.

In the drawings:

FIGS. 1A-D schematically illustrate systems for image registrationconstructed in accordance to the teachings of the present inventionsystem.

FIG. 2 illustrates a triangular pyramid with a surface checkerboardpattern which can be used as a calibration target for the system of thepresent invention.

FIG. 3 illustrates a calibration target which can be used to calibrate athermal imaging camera of the system of the present invention.

FIG. 4 illustrates a three dimensional contour model of a female breastas constructed by using the system of the present invention.

FIG. 5 illustrates a thermal image captured by the thermal camerautilized by the present invention.

FIG. 6 illustrates superimposition of thermal data on a threedimensional contour model of a female breast as constructed by thesystem of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention is of a system and method which can be usedimaging data registration and correlation of data obtained by severalimaging modalities.

The principles and operation of the present invention may be betterunderstood with reference to the drawings and accompanying descriptions.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details set forth in the following description or exemplified bythe Examples. The invention is capable of other embodiments or of beingpracticed or carried out in various ways. Also, it is to be understoodthat the phraseology and terminology employed herein is for the purposeof description and should not be regarded as limiting.

Medical imaging offers numerous imaging modality options for enablingdiagnosis of a patient. However, since images obtained by suchmodalities are not easily registered or correlated with one another,oftentimes diagnosis relies upon use of a single imaging modality or onthe ability of a physician to correlate between various imaging data.

Differences in scale, position, or in the orientation of the plane ofprojection (of a two-dimensional image) are inevitable, making freecorrelation between images nearly impossible. With certain tissues (e.g.breast) imaging registration is further hampered by deformation of thetissue which can result from the imaging technique (e.g. compression ofbreast tissue between mammography plates).

While reducing the present invention to practice, the present inventorshave devised a simple, yet effective approach for medical imagingregistration. Such an approach can be used to register imaging modalitydata taken at any time and under any settings thus enabling correlationbetween historical as well as previously non-relatable imaging data.

Thus, according to one aspect of the present invention there is provideda method of obtaining imaging data from a tissue region.

As used herein, the phrase “tissue region” refers to any region oftissue in a body, including regions defining an organ, a limb or ananatomical region of interest. Preferably, the surface of the tissueregion or a portion thereof has a contour which can be mapped via, forexample, light imaging (in the case of external, i.e. skin-juxtaposedtissue regions) or via other imaging techniques (e.g. thermal imaging inthe case of a tissue region having a surface disposed within the body).

The method of the present invention is effected by associating surfacecontour data of the tissue region in a first state with imaging datafrom the tissue region in the first state.

As used herein the phrase “imaging data” refers to data obtained by animaging approach. Such data can be in the form of two dimensional orthree dimensional data files which can be processed and presented on adisplay such as a computer screen. Such data can be ultrasound data,x-ray data, magnetic imaging data, nuclear medicine data, thermographicdata, optical imaging data, electrical impedance data, optoacousticimaging data, elasticity data, microwave imaging data and the like.

Surface contour information can be collected using any one of severalapproaches. In the case of skin-protruding tissue regions (e.g. breast),the contour of the skin can be mapped using a imaging device (e.g. a CCDor CMOS visible light camera) and projected or applied color and/orgeometrical patterns (for further description see the Examples sectionbelow). Skin contour information can also be obtained using acoordinate-measuring machine(www.en.wikipedia.org/wiki/Coordinate-measuring_machine).

In any case, once surface contour information is obtained, it isprocessed to yield a three dimensional model of the surface contour asis shown, for example, in FIG. 4. Such a contour model represents thethree dimensional appearance of the tissue region and thus can be usedas a reference map for imaging data obtained from the tissue within thetissue region.

The same tissue region is also imaged using a modality such asultrasound, MRI, CT and the like in order to obtain imaging data. Itwill be appreciated that this step of the present methodology can beeffected prior to, concomitantly with or following the step of obtainingsurface contour information.

In any case, the surface contour information and the imaging data arecollected in a manner which enables correlation between the surfacecontour of the tissue region and the data obtained via imaging. Oneapproach which can be used to enable such correlation involves the useof calibration targets. Such targets provide one or more points ofreference which can be used to map the spatial orientation of thecaptured contour and imaging data and thus calibrate the devices usedfor surface contour capture with the imaging modality device/system. Useof calibration targets is further explained in detail in the Examplessection which follows with respect to contour data obtained via visiblelight imaging.

Use of calibration targets enables either calibration of the devicesused in contour and imaging data capture, in which case, the positionand orientation of these devices can be calibrated such that they imagethe same plane and region, or alternatively, such calibration can beused to correct the contour or imaging data by mapping them to the samepoints of reference. In any case, once the devices or images obtainedthereby are calibrated, images obtained thereby are fully correlatableand can be used to provide a combined image (see FIG. 6).

For example, correlating ultrasound data with surface contour data canyield information which can be used to correlate US imaging with imagingobtained by other modalities (e.g. thermography, X-ray). A standard USimage is embodied by a two dimensional plane defined by the ultrasoundwave plane transmitted from a transmitter in the US probe. Theultrasound wave is reflected by body tissue back to a receiver, usuallyalso located within the probe. The ultrasound waves propagate in amanner determined by the location and angle of the probe; the probelocation also determines which plane in the body is imaged. Therefore,if the relationship between the angle of the probe, the direction ofwave plane emitted from it and the position of the probe is known at thetime the image is obtained, the position of the image plane with respectto the body can be determined.

Since the US probe is manually positioned on the body and since contactbetween the probe and the skin (which is required for imaging) leads tocontour deformation, the plane of an US image varies from one imagecapture to another. As a result, for each image, a different geometricstructure of the tissue is captured. Correlating such images to a singletissue region/structure can be effected by applying deformationfunctions based on deformation models. These models can be applied tospatial locations inside a tissue, in addition to spatial locations onthe surface.

By qualifying the state of a tissue region (e.g. the deformation state)using the 3-D (contour) modeling described herein and associating theposition and/or state with the imaging data (e.g. US image planes) onecan a correlate between several image planes taken at different tissuestates. Correlation can be made between a reference 3-D image and eachUS image by means of deformation conversion and thus each US image planecan be correlated with a location in the 3-D image and thus the locationin the tissue region.

Once an imaging modality is correlated with surface contour dataobtained using the present invention, any shift in the surface contourof the tissue region (i.e. shift in state) can be used to ‘correct’ theimaging data.

The calibration target must posses several characteristics to enableco-calibration of the surface and imaging data.

(i) It must be ‘visible’ to the devices used in acquisition of thesurface a contour and imaging data; e.g. the spatial reference pointsprovided on the target should be included in the data obtained thereby.

(ii) it must accurately determine the spatial location and angle(imaging or projection) of the devices.

(ii) the data obtained thereby must be correlatable with preacquired 3-Ddata (e.g. MRI data).

Medical imaging data includes a collection of data points of interest(e.g. data points representing abnormal tissue such as a tumor mass). Aphysician's main concern when comparing various imaging modalities isthe shift or movement of tissue or imaging data that occurs betweendifferent modalities or through acquisition of the same imaging data atseveral different time points. Specifically, data points of interest donot appear at the same region of an image when different planes oftissue are imaged and/or when the tissue region is subject to differentforces which can result from different imaging positions and the like.

Thus, effective registration of imaging modalities must take intoaccount tissue deformation as well as imaging planes for effective imageregistration.

By mapping the imaging data to surface contour data, the presentinvention enables effective yet simple image registration as well ascorrection of imaging data for tissue deformation and matching ofimaging modalities taken from different angles and positions.

Such correction or registration of imaging data can be further enhancedby employing a tissue deformity model which can be related to thecontour data obtained by the present approach. Such supplementarydeformity correction can be applicable in cases where the tissue imagedis deformed by the imaging device (e.g. mammography device) and thetissue within the tissue region does not exhibit uniform deformity dueto the heterogeneity of the tissue.

Tissue deformation models are well known in the art, examples includethe Finite Element method and the Linear Elasticity theory. Such modelscan be used to further enhance the correction of data point positionswithin the tissue by compensating for varying deformation of varioustissues within the tissue regions.

Such data can also be acquired by combining 3-D contour acquisitionalong with thermal imaging. This can be achieved by capturing aplurality of thermal images (preferably from different angles) from atissue region (e.g. breast) at a first state and determining thepositions of several thermal landmarks within the tissue (landmarks thatare easily recognizable and are homogenously spaced throughout thetissue are preferred). The same images can then be captured when thetissue is subjected to controlled deformation (i.e. the tissue region isin a second state) and the position of the landmarks determined again.By comparing the positions of the landmarks in both states, a map ofrelative tissue compliance (to applied force) can be constructed for theindividual imaged. Such a map can be used to model the tissue within thetissue region and predict shifts of discrete locations within the tissueregion as the tissue region deforms.

The present invention can be used to correct and register data obtainedfrom any imaging modality. One specific modality which can benefit fromthe present approach is thermal imaging.

Thermal imaging can be used to image both external and internal tissueregions; it provides a highly accurate and sensitive temperature map andthus pathological state of the tissue region of interest.

Tissues routinely imaged via thermal imaging devices include breasts,blood vessels and muscles as well as internal organs.

When applied to thermal imaging registration, the present approachenables superimposition of thermal imaging data onto the surface contourdata obtained as described herein. Such superimposition provides twobenefits registration of the thermal imaging data and such an ability tocorrelate such data with data obtained from other imaging modalities (asis described hereinabove) and a more accurate correlation between the(imaged) surface thermal data and the actual internal source of thisdata.

A thermal camera captures two dimensional images. Its output correspondsto the number of photons which strike its detectors. An electric signalis generated according to the number of incident photons. The camera‘translates’ this signal to a numerical value which can representtemperature values or relative gray level values.

In a 2D thermal image of a 3D object, pixels corresponding to slantedareas (situated in an angle relative to the camera) are lackinginformation because the infrared radiation is emitted from a larger areadetected by the camera and which is unknown to it.

In the present approach, a further connection between the valuesobtained from the thermal camera and the observed object is made,further enhancing the 2D information acquired from a standard thermalimage. As is further described herein, a thermal camera is calibratedwith a 3D imaging system and 3D and thermal images of an object areobtained (see Examples section below). Calibration allows matching ofpixel value from the 2D thermal image to the corresponding area in the3D object. This area is often larger than the size of one pixel so theinformation is matched up with a larger area, according to theinformation from the 3D object. This reflects the object's emission moreaccurately since the 3D structure is taken into account and what appearsto be a single pixel of a 2D thermal image is correlated with the truearea size thus yielding additional thermal information.

The present methodology can be carried out using a system havingsoftware and hardware components.

FIGS. 1 a-d illustrate a system for registration of imaging data whichis referred to herein as system 10. System 10 is described in contextwith breast imaging, however, it should be noted that system 10 of thepresent invention can also be used in diagnosis of other body regions,including for example, stomach, back and the like.

As is shown in FIG. 1 a, system 10 includes a projector 12 and a visiblelight camera 14. System 10 further includes a processing unit 18 whichis in communication with projector 12 and camera 14. Processing unit 18is configured for communicating a projected pattern to projector 12while acquiring and processing data captured by camera 14. In thatrespect, processor 18 stores the projection files and executes softwarewhich enables data collection and processing. To that effect a softwaresuite such as MatLab™ can be configured for processing the capturedimages in order to generate the contour model.

The components of system 10 can be included in a single housing orprovided as individually housed, yet interconnected devices.

Prior to imaging data acquisition, system 10 is calibrated using acalibration target (exemplified in FIG. 2) as is described in Example 1of the Examples section hereinbelow such that projector 12 and camera 14are co-aligned. Following calibration, system 10 is then utilized tocapture image information from the target tissue (breast 20 shown inFIG. 1 a). The image information captured by camera 14 includes aplurality of captured frames each including a different pattern 22projected on the surface of the tissue region. The captured frames arethen processed by processing unit 18 to yield 3-D contour data (anexample of which is shown in FIG. 4.

Following setup, system 10 can be utilized along with any imagingmodality to thereby enable registration of imaging data.

FIG. 1 b illustrates use of system 10 in registering image data acquiredby an ultrasound probe.

Breast contour data is acquired as described above and a breast contourmodel is generated prior to ultrasound imaging. An ultrasound probe 24is then used to scan breast tissue and acquire one or more images at oneor more US scanning planes. For each image/plane, system 10 acquiresinformation which includes the contour of the breast as deformed by theultrasound probe and the angle and thus projection plane of theultrasound probe and therefore the plane of acquired ultrasound image.

The data collected prior to and during ultrasound imaging can then beused to correlate the ultrasound images to the contour model and correctthe ultrasound images obtained to the non-deformed breast model obtainedprior to the ultrasound exam.

FIG. 1 c illustrates use of system 10 in registration of image dataacquired by an X-ray imager.

In X-ray imaging of the breast (mammography), the breast tissue iscompressed between plates and thus is deformed. Prior to breastcompression, Breast contour data is acquired as described above and abreast contour model is generated.

Following generation of such a model, breast 20 is compressed betweenmammography plates 26 and an x-ray image of the breast is then acquired.Breast 20 is also imaged using system 10 and a contour model isgenerated for breast 20 at the deformed state. The contour model cantake into account the plates and their respective positioning in orderto enhance contour modeling.

Contour model of deformed breast 20 can then be correlated with theacquired x-ray data and corrected according to the contour data acquiredprior to breast 20 compression.

FIG. 1 d illustrates use of system 10 in registration of image dataacquired by a thermal imaging device.

In such a configuration, system 10 utilizes a calibration target whichis sensitive to both camera 14 and thermal imaging device 28. Such acalibration target is exemplified in FIG. 3.

Once all the devices in the system are calibrated to the same axis(camera 14, projector 12 and thermal imaging device 28) and a contourmodel of the breast is acquired, thermal imaging device 28 is utilizedfor thermal image capture and an a combined image of thermal datasuperimposed onto contour data is generated by processing unit 18(described in detail in the examples section which follows.

Thus, data acquired by the above described imaging approaches isintegrated with the co-acquired contouring data and used for datacorrection, thus enabling correlation between various imaging modalitieswhich are acquired using system 10 of the present invention.

For example, imaging data acquired via US (along with system 10) can becorrected (e.g., adjusted in as far as imaging plane, depth etc.) usingthe 3-D contour model (generated by system 10) and the corrected imagingdata can then be correlated with similarly corrected thermal or X-raydata. Similarly, thermal data acquired while using system 10 of thepresent invention can be registered with X-ray data for the purpose of,for example, diagnosis of breast cancer.

It will be appreciated that correction of imaging data can be effectedsuch that the corrected data represents the tissue region at a singlenormalized state (for example, in the case of breast tissue such a statecan be that observed in an upright subject), or alternatively,correction can be effected such that the imaging data acquired by oneapproach is corrected to represent the tissue state (deformation state)of a tissue imaged using a second approach. For example, X-ray data ifprovided on film and thus cannot be easily manipulated can be comparedto a US image which is corrected such that the corrected US imagerepresents tissue imaged under a deformation state (e.g. compressedwithin plates) identical to that of X-ray imaging.

In any case, such co-registration of imaging data, which can be effectedmanually by simply superimposing two registered images (as softwarefiles or hard copies) or computationally, by integrating imaging dataand isolating data points of interest, enables a treating physician toverify the existence of pathologies with increased confidence thusgreatly enhancing diagnostic accuracy.

Although the present method has been described in the context of medicalimaging, it will be appreciated that the present method and system finduse in other fields including, for example, mechanical engineering andthe like.

It is expected that during the life of this patent many relevant imagingmodalities will be developed and the scope of the term imaging data isintended to include data obtained by such new technologies a priori.

As used herein the term “about” refers to ±10%.

Additional objects, advantages, and novel features of the presentinvention will become apparent to one ordinarily skilled in the art uponexamination of the following examples, which are not intended to belimiting. Additionally, each of the various embodiments and aspects ofthe present invention as delineated hereinabove and as claimed in theclaims section below finds experimental support in the followingexamples.

EXAMPLES

Reference is now made to the following examples, which together with theabove descriptions, illustrate the invention in a non limiting fashion.

Example 1 Contour Model with Superimposed Thermal Data

A model of the surface contour of a female breast was generated andutilized to map thermal data thereupon.

Material and Methods

Three dimensional contour data was obtained using a projector(Mitsubishi electronics model XD206U) and a camera (Pixelink modelPL-B741F). A thermal image was obtained using a thermal camera (FLIRmodel PHOTON OEM).

In order to obtain superimposed thermal data on a surface contour, thethermal and visible light cameras must are co-calibrated using a singlecalibration target. It is only necessary to calibrate the system oncefollowing which the location of each of the devices is fixed. Thecalibration of the cameras (video and thermal) is achieved bycorrelating pixels present in images captured by these cameras withknown spatial reference points. Similarly, the projector is calibratedby correlating projected pixels with such spatial reference points. Inorder to reconstruct the three dimensional feature of an object, imagesof patterns projected by the projector on the object are captured by thecamera and the pixels of the captured image are analyzed (as is furtherexplained hereinafter) and matched with the spatial reference points.

The spatial reference points selected for calibration can be presentedon a calibration target such as a triangular pyramid with a surfacecheckerboard pattern (FIG. 2).

Calibration of the devices is effected as follows. A point of origin isselected on the calibration target, e.g. the point protruding out in themiddle of the pyramid (FIG. 2). The reference points for calibration ofthe video camera are the square's corners; the reference points selectedfor the thermal camera are the square centers.

In the image captured, each reference point is characterized by a set ofpixel coordinates (u, v). Their spatial coordinates (x, y, z) are known,relative to the origin defined. Both coordinates can be represented byhomogeneous coordinates for simplification of calculations. Acalibration matrix, P, is constructed by correlating between the pixelcoordinates (u, v) and their spatial locations (x, y, z). This matrixsolves the following equation:

$\begin{pmatrix}u \\v \\1\end{pmatrix} = {P \cdot \begin{pmatrix}x \\y \\z \\1\end{pmatrix}}$

Its size is (3, 4) and therefore it includes 12 elements which arecomposed of the device's intrinsic parameters (pixel size, focal lengthetc.) and extrinsic parameters (device's location; angles anddisplacement compared to selected origin in space). In addition, thematrix includes perspective implementation.

Although the matrix contains 12 elements, there are only 11 unknownparameters (5 intrinsic and 6 extrinsic). As is evident from theequation above, each (x, y, z) point provides two coordinates in animage (u and v) and two separate equations, one for each pixelcoordinate. To calibrate each camera, only one image of the calibrationtarget is required. In this image, 6 pixels are selected to solve the 12equations and the 12 elements of the matrix P are extracted. In reality,more than 6 points are selected in the image to obtain higher precision.

The thermal camera is calibrated using the same process as the videocamera by correlating pixels in an image to spatial locations, solvingthe equations and constructing a calibration matrix. The differencebetween the thermal camera and the video camera is that when calibratingthe thermal camera, the pixels are selected from a thermal image and thereference points on the calibration target are thermally visible. In thepresent system, the reference points selected for calibration of thethermal camera are the square centers on the checkerboard pattern, onthe same triangular calibration target utilized for calibration of thevideo camera.

Several approaches can be used in order to make such points visible tothe thermal camera:

Using Thermoelectric Coolers (TECs) which when connected to a directcurrent source generate a temperature differential detectable by thethermal camera.

Using heat generating electrical resistors in the calibration target.

Coating the calibrating target with materials with significantlydifferent emissivity, thereby producing a pattern of dark and lightsquares.

A calibration target modified for use with a thermal imaging camera isillustrated in FIG. 3.

Calibration of the projector is also obtained by matching up its pixelswith spatial reference points. Since the projector projects pixelsrather than capturing them defining its pixels requires a more complexprocedure. Calibration of the projector is achieved by projectingspecific patterns on the calibration target and capturing all patternswith the video camera (coded light approach). By doing so, each of theprojectors' pixels is assigned a unique code. This enables correlationbetween the projector's pixels, to the images obtained by the camera.Different light codes can be utilized for this procedure. In our systemwe use the binary Gray code which consists of patterns of dark and lightstripes to perform three dimensional surface imaging [Sato and Inokuchi,J. of Robotic Systems 2(1) 27-39; 1985]. When a sequence of horizontaland vertical Gray code patterns are projected on the calibration targetand captured by the camera, each pixel attributed to the projectorpossesses its own binary code composed of ones and zeros. When the Graycode is utilized, the number of patterns required for projection dependson the number of pixels in the projector. Thus, if the projector has1024 pixels (210), 10 gray code patterns are projected so that eachpixel has its unique sequence. Now that the pixels can be identified,the procedure of corresponding them to points in the world with knownlocations, solving equations and defining the calibration matrix iscarried out while the reference points selected are the squares cornerson the calibration target (as with the video camera).

When all three calibration matrices are obtained, one for each device,they can be used to associate points in a two dimensional image with athree dimensional structure. The devices are fixed in position relativeto each other since their matrices are constructed in accordance with,amongst other parameters, their positions and angles.

RESULTS

The projector was utilized to sequentially project multiple lightpatterns onto a female breast while the camera (having a known positionwith respect to the projector) was utilized to capture reflectedpatterns. The light patterns projected was a sequence of Gray codepatterns which provide each pixel with a unique sequence of ones andzeros. These pattern points projected onto the female breast in thiscase were located in the captured image and used to extract contourinformation.

Reconstruction of three dimensional data was obtained throughTriangulation. The camera and projector were placed side by side (asopposed to one on top of the other) such that the projector projectedvertical stripes (Gray code patterns) and the triangulation wasimplemented in a horizontal manner. The basis for triangulation lies inthe triangle formed by the intersection of a camera image pixel with aplane from the projector (a plane because stripes and not dots areprojected). Each camera pixel intersects with a plane projected from theprojector at a specific point in space, on the surface of the projectedobject.

In the present system triangulation is facilitated by correlating thecamera's pixels (u, v) and their point of origin from the projectorwhich is known from the projected patterns. Each spatial was attributedto camera pixels by selecting a (u, v) pixel and examining its Graycode, as seen in the image captured by the camera. The result of theTriangulation calculation was the point's spatial location (x, y, z).

Spatial points reconstructed into three dimensional information are onlythose which are in both the camera's and the projector's field of view.

Using the above described approach, the present inventors constructed athree dimensional contour model of a female breast (FIG. 4).

Once the contour model was obtained, the thermal camera was calibratedas described above and utilized to capture thermal data from breasttissue.

Every object with a temperature above absolute zero emits radiation. Theamount of radiation emitted depends on the object's temperature andemissivity. The emissivity of a material is the ratio of energy radiatedby the material to energy radiated by a black body at the sametemperature. The human skin has high emissivity and is considered closeto 1. The amount of radiation emitted by an object increases with itstemperature and so an object's temperature can be analyzed by thermalimaging. A thermal Imager detects and displays surface temperaturesonly, which can be represented as grayscale or color images. It iscommon in a grayscale image that hot things appear whiter and coolerthings appear blacker, although this depends only on the device'ssettings.

A thermographic camera is a device which converts thermal infraredradiation emitted (and also reflected) by objects into images that canbe graphically displayed. Its function is similar to an ordinary digitalcamera which produces images by detection of visible light. Instead ofthe 400-750 nanometer range of visible light, infrared cameras operatein wavelengths from 750 to as long as 14,000 nm (14 μm) so their lensmust be transparent to infrared radiation (various cameras are sensitiveto different wavelength ranges of the infrared region and not the wholeinfrared region). Humans at normal body temperature radiate moststrongly in the infrared range at wavelengths around 10 μm. As with anydigital camera, the radiation is focused by optics onto infrareddetectors which are responsive to infrared radiation. The radiation isconverted to electrical signals which are processed and translated intoan image that can be viewed on a standard video monitor. The output ofthe thermal camera is calibrated in units of temperature.

Thermographic cameras include detectors of one of the two types; cooledor un-cooled.

Cooled thermal detectors are based on the quantum effect; a photonstrikes the detector and excites an electron with an amount of energydetermined by the photon's frequency. Infrared radiation is low inenergy so the difference between two energy levels is small and thus thedetector is highly prone to thermal noise.

Un-cooled thermal detectors are comprised of materials which respond toheat in different manners; loading of capacitor, change in resistance(bolometers), expansion of gas etc. Un-cooled detectors can be used inroom temperature but are usually less sensitive than cooled detectors.

In this example, the present system utilized an un-cooled thermal camerawith bolometers (microbolometers) as detectors. When infrared radiationstrikes the detectors, their electrical resistance changes. Thisresistance change is measured and can be processed into temperatureswhich can be represented graphically. FIG. 5 illustrates the resultantthermal image captured by the thermal camera utilized by the presentinvention.

This thermal image was then correlated with the 3-D location points(representing a surface) to obtain the (u, v) coordinates in the thermalimage which correspond to the (x, y, z) points in space. This in effectresults in projection of the 3-D surface onto the image plane of thethermal camera. Once the 3-D location points and the thermal image areco-localized to the same plane, they can be inter-associated. Usinginterpolation, every (x, y, z) 3-d location is correlated with a valuefrom the thermal image. The values in the thermal image aren't theabsolute temperatures of the object, but rather are gray levels whichrepresent the infrared flux emitted from the object and detected by thethermal camera. The resulting image now includes data points whichpossess four coordinates: (x, y, z, t). The ‘t’ coordinate refers to anumerical value in the thermal image which are added to the 3-d image ascolor or graylevels points (FIG. 6).

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims. All publications, patents and patentapplications mentioned in this specification are herein incorporated intheir entirety by reference into the specification, to the same extentas if each individual publication, patent or patent application wasspecifically and individually indicated to be incorporated herein byreference. In addition, citation or identification of any reference inthis application shall not be construed as an admission that suchreference is available as prior art to the present invention.

What is claimed is:
 1. A method of obtaining thermal data from a tissueregion comprising: (a) obtaining a first surface contour of the tissueregion while the tissue region is in a first state; (b) obtaining afirst thermal data from the tissue region in said first state andassociating it with said first surface contour; (c) obtaining a secondsurface contour of the tissue region while the tissue region is in asecond state, different from said first state, said second statecorresponding to a deformed shape of the tissue region relative to saidfirst state, said deformation being effected by a non-thermal imagingmodality; (d) processing each of said first and said second surfacecontours to provide a three dimensional model of the respective surfacecontour; (e) transforming said first thermal data into a second thermaldata associated with the tissue region in the second state; (f)obtaining non-thermal imaging data from said non-thermal imagingmodality while the tissue region is in said second state; and (g)co-registering said second thermal data with said non-thermal imagingdata.
 2. The method of claim 1, wherein said tissue region is a breast.3. The method of claim 1, wherein (a) is effected using at least onecamera.
 4. The method of claim 1, wherein (a) is effected by capturingan image of a pattern projected onto a surface of said tissue region. 5.The method of claim 4, wherein said pattern is a coded light pattern. 6.The method of claim 4, wherein said image of said pattern is processedusing a processing unit.
 7. The method of claim 1, wherein saidnon-thermal imaging modality comprises ultrasound imaging.
 8. The methodof claim 1, wherein said non-thermal imaging modality comprises x-rayimaging.
 9. The method of claim 1, wherein said non-thermal imagingmodality comprises magnetic resonance imaging.
 10. The method of claim1, wherein said non-thermal imaging modality comprises at least one ofnuclear imaging, electrical impedance imaging, optoacoustic imaging,elasticity imaging and microwave imaging.